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ABSTRACT 

Evaluation efforts related to Federal education 
programs are discussed as to steps that have been taken and the 
current evaluation plan. Evaluation functions have been centralized 
in the Commissioner's Office of Program Planning and Evaluation. The 
two major kinds of evaluation activities that are carried out are (1) 
monitoring, and (2) effectiveness evaluations. It is concluded that a 
professional evaluator should be engaged by school systems to provide 
an independent evaluation of their activities. (DB) 
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UTILIZATION OF FEDERAL FUNDING FOR EVALUATION 



There can be little dispute about the great need For hard data on the effec- 
tiveness of education programs. However serious efforts at evaluating 
Federal education programs have only a brief history. 

Prior to FY 70 only token amounts were appropriated for evaluating the more 
than $4 billion spread over 100 programs administered by the Office of 
Education . 

In FY 70, the Congress appropriated $9.5 million for planning and evaluation. 
This had not been seriously anticipated in light of the previous budgetary 
history of these requests, and since the appropriation was not passed until 
three months before the end of the fiscal year, the necessary organisation 
and technical resources to prepare and administer a comprehensive evaluation 
activity were not in place. A concerted effort was made, however, and a 
substantial number of needed evaluation studies were initiated. 

A more carefully designed evaluation plan was developed in FY 71, and an 
effort was begun to build a strong evaluation capability in OE. The FY 71 
appropriation was $11.3 million, a 207, increase over FY 70, although a number 
of directed levies and set-asides substantially reduced the amount available 
for new studies. Thus, FY 72 with approximately the same level as FY 71 
represents only the third year of systematic comprehensive evaluation efforts 
of Federal education programs. 
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It is obvious that different kinds of evaluation studies are needed to 
serve different levels of decisions. Project managers need very 
particularistic information to run their local projects better; program 
managers need data on the relative effectiveness of different programmatic 
techniques and strategies to shape the structure of the program and make 
choices among alternative approaches; and decision makers at the Agency, 
Administration, and Congressional levels need information about the over- 
all effectiveness of educational programs and expenditures. Here, the 
needs cover the spectrum of decision makers from program managers in the 
field, through top management in OE, 1IEW, and 0MI3, to the Congress. The 
in fact, has legislated that an annual report be made to them 
on the results and effectiveness of programs administered by the Office 
of Education. This has been a difficult requirement to fulfill inasmuch as 
most OE programs have never been subjected to formal evaluation. Thus, 
special emphasis has been placed in the last two years on evaluations of 
overall program effectiveness starting with the larger national programs. 
Major studies are now in progress for example to assess the impact of 
such programs as ESEA Title I, Emergency School Assistance, Vocational 
Education State Grant Programs, Teacher Training Programs, Student Assistance 
Programs, etc. 

Our goal in the next few years is to complete formal evaluations of all 
major OE programs. To this end, our FY 72 evaluation plan continues the 
major emphasis on the large scale national evaluations of overall program 
effectiveness . 
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Reformulating evaluation goals and reorganizing our evaluation efforts are 
important, but they are relatively easy tasks compared with actually per- 
forming evaluation studies, and this difficulty should not be minimized. 

The past lack of useful evaluations is not due entirely to lack of funds. 

The total complex of tasks of designing evaluations in the educational area 
which are truly policy relevant, which will withstand methodological 
scrutiny, and which can be completed in time to actually affect the pro- 
blems and decisions to which they are addressed is enoromously difficult. 

But we have made some substantial advances over the previous state of 
affairs. Thus, we have developed: 

- a comprehensive annual evaluation plan which integrates 
the needs and inputs from all the Bureaus, the Office of 
Program Planning and Evaluation, and the Office of the 
Assistant Secretary for Planning and Evaluation; 

- We have assembled more technically competent staff to 
design and direct sophisticated evaluation studies; and 

- We are establishing an improved system of getting evaluation 

| results into the larger decision making stream of planning 

and budgeting. 

i 

i 

j C . Evaluation Planning Process 

| Within those limitations the method used to develop the current evaluation 

i 

j plan has been to: 

; 1. Solicit evaluation needs and recommended studies from the Deputy 

Commissioners . 
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2. Identify Llic legislative and program objectives of all major 
programs . 

3. Assess what currently exists in terms of evaluation evidence on 
each of the programs. 

4. Take into account the utility and likely outcome of any relevant 
evaluation studies underway. 

5. Identify the remaining evaluation gaps and needs brought to light 

by this analysis and contained in the evaluation recommendations 
from the Deputies. 

6. Explore in each case a preliminary design indicating the type and 
scope of evaluation study needed. 

7. Hake a priority judgment (on the basis of the priorities noted 
above) as to which evaluation projects should be launched first 
and which left for a subsequent fiscal year. 

8. Review the selected studies with the affected Bureaus, with 
ASPE and the Special Committee of the CSSO's. 

9. Submit a final proposed plan to the Commissioner and ASPE. 

A new stop added this year for the first time has been the solicitation of 
comments and suggestions from the CSSO's through the Special Committee on 
Coordinating Educational Information. 

Since it is not possible to mount all the evaluations that might be desirable, 
in developing our plans the following kinds of studies are considered of 
highest priority: 

1. Projects which focus on the effectiveness of specific programs, 
as compared to those which emphasize general data collection. 






2. Projects which will supply information Cor Congrcasionally 
mandated evaluation reports. 

3. Projects which can he completed in time to affect upcoming 
legislative renewals . 

4. Projects which will contribute to the effectiveness of on- 
going programs by providing program managers with the informa- 
tion necessary to make internal reallocations of resources. 

I indicated earlier that since FY 1970, OE had been working very hard to 
improve its evaluation capability. Among the steps we took about a year 
ago to accomplish this, was the critical one of centralizing evaluation 
functions in the Commissioner’s Office of Program Planning and Evaluation 
(OPPE). Prior to this time, OP PE performed a coordinating function and 
each Bureau initiated and supervised evaluation of its own programs. This 
was consistent with the budget structure where each major program area 
administered by a bureau included a line item for planning and evaluation 
funds. 

We centralized evaluation for a variety of fairly obvious reasons, but first 
we distinguished between two major kinds of evaluation activities: 

1. Monitoring . The evaluation of individual projects through 
site visits, program information reports, data collections, 
and other monitoring activities, where the emphasis is on 
determining the extent to which intended program activities 
are actually in place and on assessing managerial and operational 
efficiency. 
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' 2. Effect Ivon ass Evaluatio ns . Tlic assessment of overall program 

impact and effectiveness, where the emphasis is on determining — 
usually through fairly elaborate field studies involving before- 
after measures, control groups, etc. — the extent to which pro- 
grams are successful in achieving their basic objectives. 

Under the centralization, the responsibility both for preparing the Agency’s 
evaluation plan and for designing and supervising the program effectiveness 
evaluations was assigned to Ol’PE. Monitoring activities were left the respon- 
sibility of the respective bureaus. 

The reasons for centralization were: 

1 . That it places the key executive function of determining 
the effectiveness of our various programs where it should 
be — in close relationship with the planning and budgeting 
functions and directly responsible to the Commissioner. 

2. It vests this function in an impartial, non-program-involvcd,‘ 

| technically staffed office, and thereby avoids the awkward 

j and undesirable situation of having programs evaluate them- 

selves . 

3. Such a consolidation provides a better wav t h an the previous 
arrangement for determining and im p lementing overall O F 

' evaluation prioritie s . It avoids the kind of suboptimiza- 

tion which occurs wllen the funds and initiating responsibility 

I 

for effectiveness evaluations are dispersed among the various 
bureaus. There is also the problem of responsiveness to Con- 

7 
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gressional, Departmental, and Executive Office prior i Lies . 

It was difficult to bo. responsive when the responsibility was 
not centralized. 

4. The procedural administration of evaluation funds is improved. 

The system we used to follow, with its multiple clearances, 
produced numerous misunderstandings and administrative headaches. 

5. Centralizing the responsibility for of fectivcneness evaluation 
both improves and makes more efficient the use of technical evalua- 
tion personnel . One of the major reasons the evaluation process 
lias failed so often in the Government is because of the lack of 
technically qualified people to design and monitor large scale, 
methodologically complex evaluation studies. The kind of people 
required to do this job properly — people with experience and 
advanced degrees in economics, operation research, statistics, 

and the behavioral sciences, people with skills in quantitative 
analysis, benef it/cos.t analysis, research design and psychological 
measurement — are hard to come by. The bureaus, with few exceptions 
were not adequately staffed with the kind of people required to 
design and carry out professionally acceptable evaluations. It 
would be a mistake, we felt, to try to solve this problem by build- 
ing up multiple evaluation units with this capability throughout 
the agency, even if it were possible to do so. 

6. The external image and credibility of OE's evaluation mechanism 
was not good; and we believed that if wc. were going to improve it, 
as well as improve the mechanism itself, we needed to move in the 
general direction indicated. 
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Now that T. have made the case for centralization of evaluation :i.n an 
impartial, non-program involved, technically staffed office, J hasten to 
add that we have a few exceptions. In two of our major experimental pro- 
grams, Follow-Through and Experimental Schools, there :i.s a built-in evalua- 
tion component. Program funds arc earmarked for evaluation purposes to 
assess the results of the experimental designs. However, the actual evalua- 
tions are performed by consultants or other staff not directly administering 
the programs . 

A number of other programs also provide evaluation funds for assessment at 
the local or project level to improve the management and monitoring of the 
projects. Included in this group are ESEA Title III, Supplementary Centers, 
Title VII, Bilingual Program Title VIII, Dropout Prevention, the Drug 
Abuse Education program, and the Nutrition and Health program. It should be 
understood that these evaluations do not substitute for the required national 
evaluations, but arc meant primarily to improve the local management of pro- 
jects. 

The principal Federal vehicle for helping impro^g^state and local evaluation 

capability is ESEA Title V, The total appropriated for this purpose, however, 

i 

: is only $43 million which does not go very far when divided among the 50 states. 

1 Thus, we are limited in what we can do in this direction not only by dollars 

but in manpower to provide meaningfull technical assistance, 

I have dwelt at length on the evaluation of Federal education programs, but the 
same rationale for credibility for independence of audit, and for technical 

i 

j competence applies at the local level. It is clear that under the concepts of 

i 
} 

o 
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nccountability , the public' is beginning to demand both that some attempts 
be made to evaluate educational programs and that they, the public, be 
informed of the evaluation results. As pointed out by S tenner and Webster 
in their Educational Program Audit Handbook^ "The basic notion is that 
education agencies, and those educators who are responsible for the effective 
operation of these agencies, can and should be held responsible for educational 
outcomes, e.g., children's learning. The notion, rightly or wrongly, that the 
quality of education can be enhanced by making educators responsible for their 
product is what seems to make accountability so appealing to many diversified 
groups . " 

"One management technique for implementing accountability in education is 
evaluation. Like the citizen's advisory council and performance contracting 
movements, the need and impetus for educational program evaluation comes from 
the public's anxiety for the quality of its children's education. Hence, 
restoring the public's confidence in public education necessitates a pro- 
fessional evaluation of the product and a professional report of the findings 
to the public." 

It follows that the professional evaluator should be independent of program 
administration, in a role similar to that of the fiscal auditor. The rationale 
for this is actually quite straightforward. The school administration is 
' logically assigned the function of gathering and reporting evaluation results, 
since this information is needed for internal management purposes. The cduca- 

tionnl evaluator, acting i ndependently and in a profcssioanl capacity, adds the 

^/Educational Program Audit Handbook by A. Jackson S tenner and William J. Webste 
The Institute for the Development of Educational Auditing, Arlington, Va. 

1971 . 
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credibility to administration's evaluation reports, so that interested third 
parties may freely rely upon them. The question is not one so much of objec- 
tivity as of credibility, since professional evaluators normally are objec- 
tive by nature of their training. 

The quality of the school system's internal evaluation department should not 
be a factor in determining whether or not to engage an educational evaluator. 

Just as many corporations have some of the most proficient accountants in 
the world, so do many school systems possess extremely capable internal 
evaluators. However, just as corporations engage an independent fiscal audit 
of their activities, so should school systems engage an independent educational 
evaluator of their activities. The concept of independent review lends con- 
siderable strength to a school board's call for additional resources and 
additional support. 

In conclusion, all education agencies, whether state, federal, or local, are 
required to provide some evidence of their effectiveness in order to justify 
the continued support of the public. The public T s insistent demands for proof 
of performance has led to a shift toward comprehensive evaluation. This shift 
has occurred at a time when the community's confidence in public education is 
eroding, thereby creating a credibility gap with respect to evaluation results. 
Those who utilize evaluation results (i.c., the public) may insist that, in the 
future, evaluation reports have some independent opinion or attestation 
accompanying them, just as in the financial world, where stockholders have come 
to expect- an independent fiscal audit of a corporation's activities. The opinion 
of an independent evaluator provides the best possible indication oJ; whether 
persons not associated directly with the educational process may justifiably 
rely upon those results in making important decisions regarding public educa- 
tion's future. 



